Motion-validating remote monitoring system

ABSTRACT

A method of autonomously monitoring a remote site, including the steps of locating a primary detector at a site to be monitored; creating one or more geospatial maps of the site using an overhead image of the site; calibrating the primary detector to the geospatial map using a detector-specific model; detecting an object in motion at the site; tracking the moving object on the geospatial map; and alerting a user to the presence of motion at the site. In addition thermal image data from a infrared cameras, rather than optical/visual image data, is used to create detector-specific models and geospatial maps in substantially the same way that optical cameras and optical image data would be used.

RELATED APPLICATION

The present application claims priority to U.S. Provisional ApplicationNo. 60/958,192, filed Jul. 3, 2007, and entitled MOTION-VALIDATINGREMOTE MONITORING SYSTEM, which is incorporated by reference herein inits entirety.

FIELD OF THE INVENTION

The invention relates generally to remote monitoring and securitysystems. More specifically, the invention relates to motion-validatingremote monitoring systems and methods.

BACKGROUND OF THE INVENTION

Standard closed-circuit television (CCTV) systems have long been used tomonitor locations requiring security. Such CCTV systems remotely monitorbuildings, military installations, infrastructure, industrial processes,and other sensitive locations. As real and perceived threats againstpersons and property grow, the list of locations requiring remotesecurity monitoring also grows. For example, regularly unmannedinfrastructure such as power substations, oil rigs, bridges, and so on,may now require protection through remote monitoring.

These traditional video surveillance systems may include networked videodetectors, sensors, and other equipment connected to a central site. Oneof the drawbacks to such traditional monitoring systems is that theyoften rely on human supervision to view video images, interpret theimages, and determine a relevant course of action such as alertingauthorities. The high cost of manning such systems makes themimpractical when a large number of remote sites require monitoring.Furthermore, a lack of automation in analysis and response increasesresponse time and decreases reliability.

Known automated monitoring systems solve many of these problems. Suchknown automated systems digitally capture and stream video images,detect motion, and provide automatic alerts based on parameters such asmotion, sound, heat and other parameters. However, these known automatedsystems often require large transmission bandwidths, provide onlylimited control over remote devices, remain sensitive to network issues,and struggle with accurate image and motion recognition.

Therefore, a need exists for reliable systems and methods of remotemonitoring that require limited bandwidth, while providing accuratemotion recognition and intelligent alert capabilities.

SUMMARY OF THE INVENTION

The present invention provides systems and methods to autonomously andseamlessly track moving objects in real-time over an entireremotely-monitored site. The present invention automatically andaccurately detects and tracks moving objects at a monitored site usingone or more primary detectors, without requiring secondary detectors,human operators, or other input sources to confirm or track the motion.Virtual models of each detector are created and linked to a commongeospatial map. Tying together the separate virtual models on the commongeospatial map creates a real time virtual model of an entire geographicarea including the modeling of detected objects that are located andtracked across a site. When motion is detected, a primary detector ismoved to, or trained on, the precise X-Y coordinate location of actualmotion, rather than a predefined location. Tracking occurs continuously,and in real time, across the entire monitored site, even when objectsmove out of the detection range of one detector, and into the detectionrange of another detector. Multiple detectors monitoring detected motionat a site follow priority rules provided by an on-site controller toautonomously track a moving object. After motion has been validated, auser may be notified via one or more communication methods and presentedwith a sequence of relevant motion images.

In addition, thermal image data from infrared cameras, rather thanoptical/visual image data, is used in substantially the same way thatoptical cameras and optical image data would be used. In the samemanner, when gathering temperature data from multiple detectors, usersmay be notified and presented with a sequence of relevant temperatureimages when changes occur and along with the geospatial location of thetemperature measurement.

In one embodiment, the present invention is a method of autonomouslymonitoring a remote site, including the steps of locating a primarydetector at a site to be monitored; creating one or more geospatial mapsof the site using an overhead image of the site; calibrating the primarydetector to the geospatial map using a detector-specific model;detecting an object in motion at the site; tracking the moving object onthe geospatial map; and alerting a user to the presence of motion at thesite.

In another embodiment, the present invention is a motion-validatingmonitoring system that includes a primary PTZ detector, a secondarydetector, and an on-site detector controller. The on-site detectorcontroller is adapted to receive image data from the primary PTZdetector and the secondary detector, and to use the data to create oneor more detector-specific, three-dimensional models of the image viewedby the detector, the detector-specific model being capable ofdetermining the precise location of a detected object in motion. Theon-site detector controller is further adapted to project the locationof the object in motion detected and located by each detector-specificmodel, to a geospatial map.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system diagram of a motion-validating remote monitoringsystem, according to an embodiment of the present invention;

FIG. 2 is a flowchart of the motion-validating remote monitoring systemof FIG. 1;

FIG. 3 is a flowchart of the step of creating a primary geospatial map,according to the flowchart of FIG. 2;

FIG. 4 is an image of a monitored site including two positional markers,according to an embodiment of the present invention;

FIG. 5 is a flowchart of the step of creating secondary geospatial maps,according to the flowchart of FIG. 2;

FIG. 6 is an image of a monitored site including two positional markersat a building, according to an embodiment of the present invention;

FIG. 7 is an image of a floor plan of a building with two positionalmarkers, according to the embodiment of FIG. 6;

FIG. 8 is a flowchart of the step of calibrating individual detectors toa geospatial map via virtual detector models, according to the flowchartof FIG. 2;

FIG. 9 is a diagram of a pinhole camera model as employed by anembodiment of the present invention;

FIG. 10 is a flowchart of the step of calibrating detectors to determinevirtual detector parameters, according to the flowchart of FIG. 8;

FIG. 11 is an elevation view of a set of detectors located at amonitored site and in their home positions, according to an embodimentof the present invention;

FIG. 12 is a flowchart of the step of determining fields of view,according to the flowchart of FIG. 10;

FIG. 13 is an image of a graphical user interface used for calibratingdetector field of view, according to the process of FIG. 10;

FIG. 14 is a flowchart of the step of determining zoom offsets,according to the flowchart of FIG. 10;

FIG. 15 is a detector image of a portion of a monitored site includingreference points, according to an embodiment of the present invention;

FIG. 16 is a detector overhead image of a portion of a monitored siteincluding reference points, according to an embodiment of the presentinvention;

FIG. 17 is a flowchart of the step of determining remaining virtualdetector parameters, according to the flowchart of FIG. 10;

FIG. 18 is a diagram of a camera viewing a point in three-dimensionalspace, according to an embodiment of the present invention;

FIG. 19 is a flowchart of the step of adding 3D reference pointscorresponding to viewable elevated terrain in an image, according to theflowchart of FIG. 8;

FIG. 20 is a Voronoi diagram with Thiessen polygons of the point viewedby the detector of FIG. 18, according to an embodiment of the presentinvention;

FIG. 21 a is a perspective diagram of a detector view of the point ofFIG. 18 as projected onto a three-dimensional virtual model of adetector, according to an embodiment of the present invention;

FIG. 21 b is an elevation diagram of the detector view of FIG. 21 a;

FIG. 22 is a flowchart of projecting, three-dimensional local points ofa detector model to a geospatial map, according to an embodiment of thepresent invention;

FIG. 23 is a flowchart of establishing an initial background, accordingto an embodiment of the present invention;

FIG. 24 is a diagram of several one-dimensional pixel history arrays,according to an embodiment of the present invention;

FIG. 25 is a diagram of several one-dimensional pixel history arrayswith sample pixel intensity values, according to an embodiment of thepresent invention

FIG. 26 is a flowchart of the method of updating a background, accordingto an embodiment of the present invention;

FIG. 27 a is a pixel history array at time t=0, according to anembodiment of the present invention;

FIG. 27 b is the pixel history array of FIG. 27 a at time t=1;

FIG. 27 c is the pixel history array of FIG. 27 a at time t=2;

FIG. 27 d is the pixel history array of FIG. 27 a at time t=10;

FIG. 28 is a motion image array, according to an embodiment of thepresent invention;

FIG. 29 is a filtered motion image array, according to an embodiment ofthe present invention;

FIG. 30 is a last motion image array, according to an embodiment of thepresent invention;

FIG. 31 is a filtered last motion image array, according to anembodiment of the present invention;

FIG. 32 is a screen image of an object with motion rectangles, accordingto an embodiment of the present invention;

FIG. 33 is a last motion image array with a base motion rectangle,according to an embodiment of the present invention;

FIG. 34 is a filtered last motion image array with a filtered basemotion rectangle, according to an embodiment of the present invention;

FIGS. 35 a-h is a series of diagrams of two objects in motion and theircorresponding regions of interest, according to an embodiment of thepresent invention;

FIG. 36 is a diagram of an image of an object with its object region,according to an embodiment of the present invention;

FIG. 37 is a diagram of a detector viewing an object and an objectregion, according to an embodiment of the present invention;

FIG. 38 is a diagram of an object region projected to a geospatial mapat time t, and back-projected to an earlier time t-T, according to anembodiment of the present invention;

FIG. 39 is a flowchart of a geospatial detector-specific object groupingprocess, according to an invention of the present invention;

FIG. 40 is a diagram illustrating a geospatial multiple-detector objectgrouping method, according to an embodiment of the present invention;

FIG. 41 is a flowchart of geospatial multiple-detector object groupingmethod illustrated by FIG. 40;

FIG. 42 is a screen image of a site with multiple monitored zones and adialog box for creating filters, according to an embodiment of thepresent invention;

FIG. 43 is a flowchart of a method of prioritizing detectors andvalidating detected motion, according to an embodiment of the presentinvention;

FIG. 44 is a diagram of a monitored site with prioritized detectors,according to the method of FIG. 43;

FIG. 45 is a flowchart of an alarm notification method, according to anembodiment of the present invention.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It should be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention.

DETAILED DESCRIPTION

Referring to FIG. 1, in one embodiment, the present invention is amotion validating monitoring system 100 that includes a main office 102,a first remote site 104, an optional second remote site 106, a remoteuser location 108, and Internet 110. System 100 may include additionalremote sites depending on the number of locations requiring remotemonitoring. Main office 102 connects to remote sites 104 and 106, andwith remote user location 108 via Internet 110.

In one embodiment, main office 102 includes a ScadaCam Management Server(SMS) 112, internal office user 114, computer terminal 116 withmulti-detector viewer (MCV) software 118, wireless communication device120, local area network (LAN) 122, main office intranet 124, andfirewall 126. Wireless communication device 120 may be a mobiletelephone, pager, or other wireless handheld communication device.Computer terminal 116 connects to SMS 112 via main office intranet 124and LAN 122. In other embodiments of main office 102, computer terminal116 connects directly to SMS 112. Main office 102 connects to Internet110, that may be a public or private internet, through firewall 126.

In one embodiment, remote sites 104 and 106 each include one or moresecondary detectors 128, one or more primary pan-tilt-zoom (PTZ)detectors 130, raw data 132 comprising detector images 129, on-sitedetector controller (ODC) 134 with database 136, processed site data138, and optional structure 140. Secondary detectors 128 include fixedcameras that may be digital or analog video cameras that provide videooutput of all or a portion of a remote site. Secondary detectors 128 mayinclude single-, or limited-function detectors such as motion detectors,fence shakers, door and window sensors, and so on. Generally, secondarydetectors 128 provide a single view of all or a portion of a remote site104 or 106, and do not pan or tilt.

Primary PTZ detectors 130, include PTZ cameras and other detectors thatprovide pan, tilt, and zoom capabilities, and may be controlled by anumber of remote or local sources, including, but not limited to, ODC134. In some embodiments, primary PTZ detectors may not all be capableof panning, tilting, or zooming.

Although embodiments of system 100 discussed herein include the use ofsecondary detectors 128, unlike most previously known systems, system100 of the present invention does not require the use of secondarydetectors 128. Previously known motion-tracking systems typically relyon secondary detectors, usually fixed optical cameras, to firstly detectmotion. Then, based on the detection viewed by a secondary detector, aPTZ camera is panned, tilted, and/or zoomed to a predefined coordinatewithin the field of view of the secondary camera that detected motion toview the purported motion. Motion is confirmed using the PTZ camera, andonce the moving object moves out of view of the secondary camera, thePTZ camera no longer tracks. Accordingly, and unlike system 100 of thepresent invention, a secondary detector working in conjunction with aprimary PTZ detector is required to detect motion. System 100 does notrequire a secondary detector, and is capable of moving immediately to alocation of detected motion, rather than a predetermined location.

Secondary detectors 128 and primary PTZ detectors 130 connect to ODC 134using wired, wireless, fiber, USB, or other appropriate technologies. Inother embodiments, remote site 104 or 106 may include other monitoringand sensing devices, such as motion detectors, sound recorders, door orgate switches, perimeter security devices, and so on, in addition tovideo cameras.

ODC 134 coordinates and provides a number of different functions such asan integrated digital video recorder (DVR), motion detection, motionfollowing automation engine, communications managers, self-healthmonitors, local I/O management, while processing all inputs, raw data132, such as image data, and requests, and ultimately producing anoutput of videos and images 129, or processed site data 138. Motionfollowing entails directing one or more detectors to follow objects inmotion according to priority rules and detector availability rules asdescribed further below. ODC 134 includes electronic components such asprocessors, memory devices, and so on, especially adapted to implementthe algorithms and methods described below. Such algorithms and methodsmay take the form of software modules, each adapted to perform one ormany of the functions describe herein.

ODC 134 may connect to Internet 110 via wired or wireless technology. Insome embodiments, ODC 134 may connect to Internet 110 using DSL,broadband, fiber optics, or other high-speed connection. However,high-speed connectivity may not be available at many remote sites 104 or106, so slower speed connectivity, for example, narrowband transmissionover a telephone network, may be used.

Remote sites 104 and 106 may also include structures 140 to be monitoredby system 100. Structure 140 may be a building, power substation,industrial equipment, or any other item requiring monitoring. In otherembodiments, remote site 104 may be a geographical area that does notinclude a structure 140.

In one embodiment, remote user location 108 includes a remote user 115,computer terminal 117 with multi-camera viewer (MCV) software modules118, and wireless communication device 121. Computer terminal 117connects to Internet 110, linking it to the other devices in system 100.Remote user 115 gains access to system 100 for viewing detectors 128 and130, configuring the system, and accessing reports. Multi-camera viewersoftware generally includes the capability to view images communicatedfrom ODCs 134 at remote sites 104, 106.

In general operation, detectors 128 and 130 monitor remote sites 104 and106, feeding video images or raw data 132 to an ODC 134 at each site.ODC 134 manages and controls detectors 128 and 130, directing them totrack objects in motion, processes raw data 132, and communicatesprocessed site data 138, which may include video and report information,to main office 102, and remote user sites, via Internet 110. Data 138 isreceived by SMS 112 and made accessible to internal office users 114 orremote users 115 running MCV software 118 on their respective computerterminals 116 and 46. Alternatively, SMS 112 may wirelessly transmitselected information, such as alarms in e-mail or text messagingformats, to a user's wireless communication device 120.

In one embodiment, because all resources attach to SMS 112, networkinginstallation and security becomes easy. Not having to route all usersout to remote site devices maintains security. Also, because of thistopology, remote users 115 are attaching to SMS 112 usually over ahigh-speed network, providing very quick access to videos and reportinformation. Conversely, if remote users 115 were to attach directly toremote site devices to obtain videos and information, the slow-speedconnection typically used to connect the remote devices to an internet,would dramatically slow access to the information, despite a user'shigh-speed connection to an internet.

As such, the present invention at least delivers a reliable,cost-effective solution for remote security and monitoring, eliminatingthe need for round-the-clock human supervision of remote sites 104 and106. Further system 100 provides automated unattended surveillance ofremote sites 104 and notifies users 115 and 114 within seconds of analarm-level event through a monitoring application, e-mail, or textmessage. After an alarm notification, a user 115 or 114 may retrievelive streaming feed or view recently recorded high-resolution digitalvideo or images from remote sites 104 or 106.

As will be described in further detail below, and unlike previouslyknown systems, system 100 creates and uses mathematical,three-dimensional virtual models of each detector 128, 130, referencedas detector-specific models, or detector models 160 below. Thesedetector-specific models 160 are linked to a single master model of site104, referenced as geospatial map 154 below. Detector models 160simulate all physical awareness of each detector 128, 130 andautonomously track any motion activity that any of the detectors 128,130 detect. All motion activities are placed on to the mastermathematical model geospatial map 154 for seamless, real-time,detector-to-detector tracking of objects in motion.

A key difference between system 100 and known systems is that knownsystems solely use the actual detector to keep track of objects inmotion. Once an object leaves the view of a detector of a previouslyknown monitoring system, the system no longer knows where the object islocated. The systems and methods of the present invention only use whateach detector 128, 130 sees as an intelligence gathering vehicle,regarding object size and x-y-z coordinate location, and places it ingeospatial map 154. The tracking actually goes on inside the geospatialmodel, geospatial map 154, a scaled simulation, rather than at thereal-life detector 128, 130.

After detector-specific model 160, a three-dimensional spatialenvironment (x, y, z) of a remote site 104 is built, the 4th dimensionof time is added where the real world object's size, speed andtrajectory of an object 166 is mathematically simulated using a viewfrom each detector 128, 130, at different angles from the object.Trajectories of objects 166 being tracked can be plotted and pathsmathematically can be forecasted.

Moving objects 166 can pass behind obstructions on site 104, blocking itfrom the view of detectors 128, 130 but system 100 can do detectorhandoffs in the geospatial model to seamlessly continue tracking as anobject 166 leaves the view of one detector and comes into the view ofthe next one. This can occur only because the tracking and hand offs aredone inside the model as opposed to the real world field detectors.

Further, a broad unique feature of motion-validating security system 100is that it is able to track objects 166 in motion, with a changingbackground, using multiple primary PTZ detectors 130, and optionallywith secondary detectors 128.

As described below in further detail, the present invention alsoincludes a method of managing, grouping, and identifying all objectspresent within the spatial environment. The information is againfiltered through algorithms to further classify the metric informationas a whole. Physical width in scale, height in scale, color histogramcomputation, speed parameters, current spatial location, and projectedtrajectory all associated to a motion alert zone are processed. Allobjects from all detectors 128, 130 in the area are trackedindependently and objects 166 that pass the algorithms filtering willnow be considered of high importance, or validated objects. As describedbelow, validated objects are those objects that ultimately are projectedto geospatial map 154.

The present invention is also a determination process that occursagainst all of the information and variances of objects 166 to make adetermination whether each validated object is a threat and needs to bereported to a user. In addition to the physical width in scale, heightin scale, color histogram computation, speed parameters, current spatiallocation, and projected trajectory the object is placed with exceptionto user identified motion alert zones to determine the appropriatereporting scenario.

Referring now to FIG. 2, the depicted flowchart illustrates thehigh-level operations of an embodiment of system 100 as applied to amonitored remote site 104, or a monitored remote site 106.

Initially, secondary detectors 128 and primary PTZ detectors 130 areinstalled, along with other components of monitoring system 100 atremote site 104, as indicated by Step 150 of FIG. 2. At Step 152, aprimary two-dimensional geospatial model, or map, 154 of the site beingmonitored, remote site 104 in this example is created. If at Step 156 itis determined that additional, secondary geospatial maps 154 are needed,these are created per Step 157. Additional, secondary geospatial maps154 may be of buildings, structures, or other particular areas ofinterest within remote site 104.

After creating one or more geospatial maps 154 at Steps 152 and 157,secondary and PTZ detectors 128 and 130, respectively, at Step 158, arecalibrated to geospatial maps 154. This step includes creating adetector-specific virtual, three-dimensional (3D) model 160 for eachdetector 128, 130. Detector model 160 is a three-dimensional earth gridmodel of the points on the earth being seen by each detector 128, 130,and includes construction of images viewed or detected with eachdetector 128, 130. The basis behind this unique method is that a scaledmathematical model of the real world view of a detector 128, 130 isbuilt and used for simulation of objects 166 located on, or passingacross, a site 104. Details of Step 158, including the creation ofdetector model 160 will be described in further detail below.

At Steps 162 and 164, system 100 monitors remote site 104, continuouslychecking for any objects 166 in motion within the bounds of remote site104. If an object 166 in motion is detected, the exact location ofobject 166 is determined, and object 166 is tracked, as indicated atSteps 168. “Tracking” in part includes ODC 134 directing one or moredetectors 130 to automatically and continuously view object 166. At Step170, system 100 determines object 166 properties, including height,width, direction and speed, among others.

The details behind each process Step 150 to 170 are described in moredetail below. Further, although each step is presented in a particularsequence for the purpose of explaining the unique features andoperations of the present invention, the sequential order of each stepmay vary from the order presented.

As indicated above, and with reference to Steps 150 to 166 of FIG. 2, amathematical virtual 3D detector model 160, is built for each detector.Models 160 are used to project as accurately as possible, points from adetector image frame to a three-dimensional x,y,z coordinate in thecontext of a user-supplied and accurately scaled overhead map, diagram,or drawing. The model is also used to perform the reverse projection.

With respect to Step 150, secondary detectors 128 and Primary PTZdetectors 130 are initially installed at a remote site 104. The numberof detectors 128 and 130 installed may vary from site to site, dependingon considerations that include remote site 104 geographic size, numberand degree of site 104 elevation changes, existence and quantity of site104 portions requiring additional monitoring, and so on. Further, anycombination of secondary detectors 128 and primary PTZ detectors 130 maybe used. Although both secondary and primary PTZ detectors 128, 130 maybe installed at a typical site 104, some remote sites may only includesecondary detectors 128 or primary PTZ detectors 130.

Further, the term “detector” as used in the present invention includesdevices, such as cameras, used to capture images using light in thevisible spectrum, such as an optical video camera, but may also includedevices capable of capturing images using wavelengths elsewhere in theelectromagnetic spectrum, including infrared, thermal, and so on.

The term “detector” also generally includes other detectors or sensorscapable of collecting data that may correlate to, and be used to detect,an object in motion at a monitored remote site 104.

For example, in one embodiment, system 100 includes optical fixed andPTZ video cameras as detectors 128, 130. Pixel data describing visiblespectrum images are captured and used to create detector-specific models160 and geospatial map 154, and are used to track motion. In anotherembodiment, system 100 uses thermographic cameras as detectors 128, 130.In such an embodiment, system 100 uses thermal image data, rather thanoptical/visual image data, to create detector-specific models 160 andgeospatial map 154, in substantially the same way that it would, usingoptical cameras and optical image data.

Referring now to Step 154 of FIG. 2, and to FIGS. 8 to 16, geospatialmap 154 of remote site 104 may now be created. Geospatial map 154 isessentially a two-dimensional scaled overhead image of remote site 104overlayed with a coordinate system defined in terms of pixels anddistance units. Every point, or location, shown on geospatial map 154 isdefined by a geo-coordinate (GC). A GC is defined by an “x” coordinateand a “y” coordinate, each corresponding to actual distances from anorigin reference point, referenced by (0,0), and corresponding to unitsof measure such as feet, meters, or other, so that any specific GC maybe designated (x,y). As will be described further below, each portion ofan image 129 captured by a detector 128, 130 may be correlated to acorresponding GC on geospatial map 154.

Accordingly, to begin defining geospatial map 154, horizontal andvertical scales defining the number of pixels per geospatial map 154unit are determined.

As will be explained further below, points defined or identified withindetector specific models 160 also use x,y,z coordinates to definelocations of points, or objects. However, the initial x,y,z coordinatesin detector-specific models, though eventually scaled, are specific andrelative to each detector. Each detector location within its model 160is considered the origin, and the home direction of each detector 128,130 is considered the directional north for the associated model 160. Assuch, detector-specific horizontal and vertical offsets must becalculated to project detector-specific locations to the real-world,geospatial x,y coordinates of geospatial map 154. Further, a direction,or angle of rotation offset is also required to transform directionalnorth in detector-specific models to line up with wherever “North” is inthe overhead image.

With respect to Step 150 of FIG. 1, creating a first or primarygeospatial map is depicted and described using the flowchart of FIG. 3.

At Step 174, digital overhead image 176 is loaded into system 100.Referring also to FIG. 4, in one embodiment, overhead image 176 may be asatellite image, or other overhead image, of remote site 104. In otherembodiments, overhead image 176 may be an elevation drawing, floor plan,or other such drawing or image. Digital overhead image 176 willinherently be defined by a total number of pixels, as well as a pixellength Lp, and a pixel height, Hp.

At Step 178, a user 114 at main office 102, possibly an administrator,or a remote user 115 at a remote user location 108, viewing overheadimage 176 at a computer terminal 116, uses a mouse and cursor to inserttwo markers 180 and 182 onto a displayed version of overhead image 176,previously loaded into system 100 at Step 174. The distance betweenthese markers is either an actual measured distance known to the user,or may be calculated by system 100 using known GPS coordinates of thelocations corresponding to each marker 180 and 182.

At Step 184, user 114 enters the actual distance in terms of feet ormeters, or other units of measure. Alternatively, at Step 184 a, user114 enters a pair of GPS coordinates corresponding to the marker imageson overhead image 176.

At Step 186, system 100 calculates a horizontal and vertical scale foroverhead image 176 by dividing the number of pixels separating markers180 and 182 on image 176, by the actual measured distance. For example,if the distance between markers 180 and 182 is 664 feet, and the numberof pixels separating markers 180 and 182 is 500, a scale of 0.753 pixelsper foot is determined. Alternatively, at Step 186 a, system 100 usesprovided GPS coordinates to determine horizontal and vertical scales.

In some embodiments, a user specifies directional north on geospatialmap 154. In other embodiments, directional north is defined by defaultas an image bottom-to-top direction.

As such, geospatial map 154 of remote site 114 has been created bycalculating a pixel/distance scale and applying a set of geospatialcoordinates (GCs) to overhead image 176.

System 100 also provides the ability to use multiple overhead images fora single site 104 to create multiple geospatial models 154, as indicatedat Step 157 of FIG. 2. User 114, 115 is allowed to switch among thesegeospatial maps 154 on a viewer application. An example would be to usea primary satellite overhead image 176 to create a primary geospatialmap 154 a of a large campus, parking lots, and paths for outsidesurveillance, as well as use an interior floor image 188 andcorresponding geospatial map 154 b of a building on campus to showindoor surveillance.

If detectors 128, 130 are positioned and calibrated to primarygeospatial map 154 a, they will look aligned and scaled correctly onthat primary geospatial map 154 a. However, the interior floor image maybe of a different scale, most likely do not cover the same area, and mayeven have “North” pointing in a different direction than the satelliteimage. Therefore, the present invention provides a process ofcalibrating a secondary overhead image 188 to an already establishedgeospatial map 154 a, used to create a secondary geospatial map 154 b.This process takes advantage of the previously definedhorizontal/vertical scale, offsets and direction parameters describedabove.

Referring to FIG. 5, Steps 190 to 212 describe the process ofcalibrating a secondary geospatial map 154 a.

Referring also to FIGS. 6 and 7, two new markers, 192 and 194 have beenadded to geospatial map 154 a via user 114, 115. Markers 192 and 194mark two corners of a building of interest 196 located on geospatial map154 a and image 188. A new geospatial map 154 b is derived fromsecondary overhead image 188, in this case, a floor plan of building ofinterest 196, and from information associated with geospatial map 154 a.

Referring again to FIG. 5, at Step 190, secondary overhead image 188 isloaded into system 100. At Step 198, system 100 loads stored image 176of geospatial map 154 a, and displays both geospatial map 154 a andsecondary overhead image 188. At Step 228, while having both previouslycalibrated geospatial map 154 a and new, image 188 onscreen, user 114drags marker 192 to a point on geospatial map 154, and then drags thesame marker 192 to the same point on overhead image 188. At Step 202,additional markers, including marker 194, may be correlated betweenimages.

Parameters are then calculated for secondary image 188 that willtransform real-world, geospatial x,y coordinates (GCs) of the twomarkers 192 and 194 already established on primary image 176 andgeospatial map 154 a into GCs that equal where the markers have beenplaced in secondary image 188. This is done at Steps 210 to 212.

At Step 204, and also referring to FIG. 7, a secondary image 188rotation angle is calculated by subtracting angle A_(s) of geospatialmap 154 b coordinates marker 192 to marker 194 from angle A_(p) ofsecondary overhead image 188 coordinates marker 192 to 194.

At Step 206 a scale is calculated by dividing pixel distance betweenmarker 192 and marker 194 on secondary image 188 by the previouslymeasured or calculated distance between them.

At Step 208, GCs of marker 192 are rotated an amount equal to therotation angle calculated in step 204 and then multiplied by the scalefactor calculated in step 206.

At Step 210, subtracting marker 192 GCs on primary overhead image 176from the result calculated in step 208 to yield a distance offset fromprimary overhead image 176 to secondary image 188. The distance offsetcan then be taken into account when mapping image 129 points tosecondary geospatial map 154 b.

The above described process yields a second geospatial map 154 bcorrelated to geospatial map 154 a such that pixels or points from animage 129 may be projected to a correct location, defined by a single,common GC on either geospatial map 154 a or 154 b. The reversetransformation is also possible.

Once geospatial maps 154 have been created per Steps 152 to 157 of FIG.2, the next step, Step 158, calibrating individual detectors togeospatial maps 154 using detector models 160, Step 158 of FIG. 2, maybe implemented as described and depicted in the flowchart of FIG. 8.

Referring to FIG. 8, calibrating detectors to geospatial maps 154includes Steps 214 to 220. Steps 214, 216, and 218 essentially describethe creation of detector-specific, 3D virtual models 160, where eachmodel 160 may be thought of as a collection of two- andthree-dimensional reference points or coordinates, a collection ofmeasured and virtual detector parameters, and a 3D wire frame model ofthe portion of site 104 appearing within an image 129 of the detector128, 130. In the context of the present invention, two-dimensionalrefers to a coordinate that has a z coordinate of zero.

At Step 214, each detector 128, 130 is calibrated to determine itsactual and virtual detector parameters, and a number of two-dimensionalreference points are determined. Step 214 is described in further detailbelow with reference to FIGS. 8-17.

At Step 216, a number of three-dimensional reference pointscorresponding to viewable elevated terrain or structures at site 104 aredetermined. Step 216 is described in further detail below with referenceto FIGS. 18-19.

At Step 218, a three-dimensional wire frame model simulating points ofimage 129 is created. Step 218 is described in further detail below withreference to FIGS. 20-21.

At Step 220, detector models 160 are employed as needed during themotion tracking processes to project or correlate points viewed inimages 129 to geospatial maps 154, as described in farther detail below.

With respect to Step 214 and detector calibration, each detector 128,130 may be initially modeled as a pinhole camera, as depicted in FIG. 9.In the pinhole camera model, the basic mathematical relationship betweenthe coordinates of a three-dimensional point and its projection onto animage plane are well known. The camera aperture, or focal point, isdescribed as a point and no lenses are considered in the model.

Further, each detector 128, 130 is said to capture an image or frame 129that can be defined by a matrix of pixels, as known by thoseskilled-in-the art.

Still referring to FIG. 9, the following terms are used throughout todescribe the optics and the poses of detectors 128, 130:

Horizontal Field of View (A)—Describes the angle that detectors 128 and130 are able to “see” horizontally. For a primary PTZ detector 130 withvariable zoom, multiple horizontal field of view (FOV) measurements maybe stored for incremental zoom levels. For example, horizontal FOVmeasurements may be stored for every 10% of zoom level between 0% and100% of the total zoom range for eleven measurements total.

Vertical Ratio—Describes the ratio of the horizontal FOV to the verticalFOV.

Zoom X and Y Offsets—On a variable zoom detector 130 or camera, zoomingin or out can cause the focal point in the center of the frame to shifthorizontally and/or vertically due to the mechanical movement of lenses.The present invention may store and utilize a zoom offset for certainzoom level increments to adjust the pan/tilt direction that detector 130is facing in order to compensate for this shifting focal point. In oneembodiment, an offset for every 10% zoom level is stored and utilized.

Detector Rotation Angle (B)—Detector Rotation Angle B is defined as theangle the detector 130 head deviates from a level plane If looking atthe horizon through the view of the camera, points selected across thehorizon should ideally project out to a flat plane in a lineperpendicular to the angle that the camera is facing on an overheadview. In the case of where the horizon appears slanted due to detectorhead rotation, this parameter is used to mathematically rotate pointsback to a level position before projection.

Lens Distortion—Almost all lenses have some type of distortion thatcauses straight lines to appear bowed in or bowed out from a centerpoint of the image. In some embodiments, system 100 corrects for thisdistortion to achieve proper projection. Because lens distortion is notalways centered around the very center of the frame, distortion centerpoint may be identified, stored in x,y screen coordinates, and utilized.

Tilt Directional Angle (C) and Tilt Displacement Angle (D)—Becausedetectors are rarely mounted perfectly level, whether up on a pole or ontop of a building, tilt directional angle is used to specify thedirection (0-360 degrees) of where the detector is tilted. Tiltdisplacement angle D defines of the degree of tilt at that direction.

Geospatial X Offset (E) and Y Offset (F)—Describes where on thegeospatial map detector 128 or 130 is located in units applicable togeospatial map 154, such as feet or meters.

Geospatial Direction Delta (G)—The angle difference between “North” ongeospatial map 154 and the detector's home position. “North” may referto the commonly understood cardinal direction North, or may refer to adirection corresponding to a predefined direction as indicated ongeospatial map 154.

Detector Height (H)—Distance of base of detector 128, 130 up todetector's focal point.

Referring to FIG. 10, Step 214 of FIG. 8 may be broken down into aseries of Steps 224 to 234 as depicted and described.

First, at Step 224, an individual detector 128 or 130 is selected forcalibration. At Step 226, if the selected detector is a Primary PTZdetector 130, fields of view and zoom offsets are determined per Steps228 and 230, respectively. If the selected detector is a secondarydetector 128, then the process proceeds to Step 232, associatingtwo-dimensional points from an image 129 to geospatial map 154.

In the case where the selected detector to be calibrated is a primaryPTZ detector 130, at Step 228, detector 130 fields of view aredetermined according to FIGS. 11-13.

Referring specifically to FIG. 11, in the depicted embodiment, remotesite 104 includes three primary PTZ detectors 130, primary PTZ detectors130 a, 130 b, and 130 c. Using a two-dimensional Cartesian coordinatesystem, a centerpoint of remote site 104 is defined at (0,0), andcorresponding to the geospatial map 154 origin, and detectors 130 arelocated at (−120, 50), (120, 20), and (−80, −120), respectively.Although any number of units may be used, in this embodiment, units inthis case correspond to feet, as used in geospatial map 154. DirectionalNorth is indicated by arrow 236, and corresponds to 0°. “Home” positionsof each detector 130 are defined in this embodiment as 135°, 270°, and180°, respectively, are rotational offsets to be considered when mappingpoints from model 160 to geospatial map 154.

Once each detector 130 position is defined relative to site 104, Step228 of FIG. 10, a field of view calibration process for each detector128 and 130, is implemented. A rotation calibration process for primaryPTZ detectors 130 is also implemented.

Referring to FIGS. 12 and 13, a detector 130 field of view and rotationcalibration process is described in Steps 238 to 254 and in thegraphical user interface (GUI) image 256, respectively. The field ofview and rotation calibration process yields detector parameters ofhorizontal FOV, vertical FOV, and vertical ratio, for each zoom level ofa detector 130. This information is used in part to define a detectormodel 160 of a detector 130.

To start, at Step 238, a user moves primary PTZ detector 130 so that itis centered on a corner or other type of feature with definition in theimage 129 a frame, which defines a reference point R. At Step 240, viaeither a manual or automated process, detector 130 shifts left by half aframe of reference point R and captures an image 129 b, leavingreference point R at a right-most side of image 129 b. At Step 242, thedetector is then shifted right by half a frame of the reference pointand captures a second image 129 c, leaving reference point R at aleft-most side of image 129 c.

As indicated at Step 244, and as depicted in FIG. 6, the reference imageand the two shifted images 129 b, c, are displayed to user 114, 116 atGUI image 256. At Step 246, a user clicks on reference point R in bothadjusted images 129 a and b.

The number of pixels associated with each image 129 is known in advance,and is a function of detector 128, 130. As will be understood by thosefamiliar with digital imaging technology, a digital image displayed on amonitor or screen may be defined by a pixel matrix, such that any given“point” on a displayed image is associated with a pixel, and a pixellocation coordinate. For example, a JPEG-formatted image displayed on ascreen may be 700 pixels wide (“x” direction) by 500 pixels tall (“y”direction), with, for example, the extreme upper left displayed pixelcorresponding to pixel coordinates (0,0), and lower right displayedpixel corresponding a pixel coordinates (699, 499). Such relativecoordinate data corresponding to individual pixels, or locations on animage, may be captured using known technology.

In the present invention, image 129 is displayed at terminal 116, 118and viewed by a user 114,115. As a user 114, 115 follows the stepsdescribed above, a left-most pixel location or coordinate is capturedwith one mouse click, followed by a right-most pixel location with asecond mouse click. Accordingly, a horizontal image length as measuredin pixels is determined for a given image 129. Knowing the movement ofdetector 130 in degrees of rotation, a horizontal FOV can be defined interms of pixels as indicated at Step 248.

At Step 250, the process steps of 238 to 248 are repeated in a verticalcontext to determine a pixel-defined vertical FOV. At Step 252, avertical ratio is calculated using the horizontal and vertical FOVs.

At Step 254, the zoom increment is increased by a step, and steps 238 to252 are repeated to define a horizontal FOV for each periodic zoomincrement for each detector 130. In one embodiment, the zoom incrementschange in 10% increments, so that eleven different horizontal FOVs arecalculated. The vertical FOVs are calculated for each zoom levelincrement by dividing the vertical ratio determined earlier. Todetermine an approximate camera head rotation, the angle from theleft-most pixel location to the center of the left image, and the anglefrom the center of the right image to the right-most pixel location areaveraged together.

Referring to FIG. 14, a zoom offsets calibration process, Step 230 ofFIG. 10, is also applied to primary PTZ detectors 130.

In the same fashion as the field of view calibration, at Step 258, user114, 115 moves detector 130 to center on a reference point at a zoomlevel of 0%. At Step 260, using either a manual or automated process,detector 130 is zoomed in one increment. In one embodiment, the zoomincrement may be 10%. At Step 262, user 114, 115 then adjusts theposition of detector 130 to center on the same reference point. In somecases, due to the particular properties of detector 130, the detectormay or may not need to be moved.

At Step 264, an x and a y offset, defined in numbers of pixels, are thencalculated for the current zoom level by taking the difference betweenthe current (x,y) position of the detector and the reference position ofthe detector at 0% zoom.

At Step 266, if the zoom level is at 100%, zoom offsets for each zoomincrement have been determined, and the process is complete. If the zoomlevel has not reached 100%, Steps 260 to 266 are repeated until all zoomoffsets are determined.

The zoom offset parameters for each primary PTZ detector 130 are thensaved in memory and used to create 3D virtual model 160.

Referring again to FIG. 10, after a secondary detector 128 is selectedat 224, or after primary PTZ detector 130 fields of view and offsets arecalculated via Steps 228 and 230, Step 232 is implemented.

Step 232 associates a finite number of “two-dimensional” (2D) points onimage 129 to corresponding points on geospatial map 154. In doing so, aset of reference points with corresponding, unique, GCs is established,and used to create a 3D model 160 that can extrapolate further pointsand their coordinates.

Referring to FIGS. 15 and 16, Step 232 begins with user 114 identifyingat least five points A-E that appear in both detector image 129 andgeospatial map 154. Points A-E should generally be at the same altitudebecause they will eventually define a base plane with a z-coordinate ofzero|_([A1]).

User 114, 115 operating a terminal 116, 117, simultaneously views bothdetector image 129 and geospatial map 154. In a manner similar to theprocesses described above, uses a mouse and cursor to drag and drop eachof the five points A-E from image 129 on to geospatial map 154. It willbe appreciated that other methods and techniques may be used to matchpoints from image 129 to geospatial 154. Further, in some cases, more orless than five matching points may be used, depending on site 104characteristics and desired accuracy.

After the point-matching detector calibration process, the GCs of pointsA-E appearing in image 129 are now defined, i.e., x- and y-coordinatescorresponding to geospatial map 154. The GCs of these points A-E arethen used as input into an algorithm of the present invention thatcreates a unique virtual 3D detector model 160 for each detector128,130.

One algorithm widely used in the computer vision industry to createvirtual detector models, which includes defining virtual detectorparameters, is disclosed in a paper authored by Roger Y. Tsai andentitled “A Versatile Camera Calibration Technique for High-Accuracy 3DMachine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses,”which is hereby incorporated by reference. It allows an input oftwo-dimensional screen coordinates and actual three-dimensional “world”or site coordinates, and outputs detector parameters, such as FOV,vertical ratio, detector coordinates, including height, etc., requiredto define a 3D virtual model. However, the methods described by Tsaionly apply to an immovable (fixed) detector, whereas the approach of thepresent invention as described below works on either a secondarydetector 128 or movable primary PTZ detector 130.

Referring to FIG. 17, an improved algorithm for determining virtualdetector parameters, according to Step 234 of FIG. 10, is depicted anddescribed. It will be understood that detector parameters refers atleast to camera location, horizontal field of view, camera height,camera tilt direction/tilt delta, lens distortion, and camera headrotation.

At Step 268, detector parameters are initially set to default levels.For primary PTZ detectors 130, certain previously determined parametervalues, for example, horizontal and vertical fields of view for eachzoom step, vertical ratio, and so on, may be used to set initialparameter values. For other primary PTZ detector 130 and secondarydetector 128 parameters, the algorithm of FIG. 17 will iterativelycalculate best values for such parameters.

At Step 270, maximum and minimum values representing a span or range foreach parameter is selected and input to the algorithm. Initially, thesevalues may be set to absolute theoretical maximums and zero, may be setaccording to estimated values based on known detector properties, or maybe estimated based on previous experience. The initial span is notcritical as these may be adjusted iteratively based on algorithmresults.

At Step 272, an increment, or step, is selected for each parameter. Aswith the span selected above, the initial step value selected is notcritical and may be adjusted to achieve improved results if needed.

At Step 274, using current detector parameter values (initially set atdefault values), the previously identified image 129 point set A-E isprojected to a virtual flat plane using known techniques such as thosedisclosed by Tsai, thereby creating a projected point set A′-E′ with acorresponding set of projected x,y coordinates.

Next, at Step 276, a known alignment algorithm is used to move, or alignthe projected point set A′-E′ to a region of the known point set A-E.Because it is not known what the location and direction of detector 128,130 is with respect to geospatial map 154 (for example, refer back toFIG. 11), a point set alignment algorithm is run to move or alignprojected points A′-E′ to the GCs of known points A-E, as closely aspossible. This alignment algorithm is detailed in the paper authored byDivyendu Sinha and Edward T. Polkowski and entitled “Least SquaresFitting of Two Planar Point Sets for Use in Photolithography OverlayAlignment,” which is hereby incorporated by reference in its entirety.If the x,y coordinates, or locations, of newly aligned point set A′-E′precisely, or closely matched the GCs of known point set A-E, the errorwould be zero or small, indicating that the current set of virtualdetector parameters provides a good model for the actual detector 128,130 providing image 129. However, several iterations of the algorithmtypically are required before acceptable detector parameters aredetermined.

At Step 280, if the error determined at Step 278 is smaller than anypreviously determined error, the current detector parameter values arestored at Step 282. Otherwise, at Step 284, the detector parametervalues are checked to see if the maximum values have been reached, andif not, at Step 286, the detector parameter values are increased by thestep values identified previously at Step 272, and Steps 274 to 284 arerepeated, until the detector parameters that provide the most accurateprojection of points A-E are determined and stored.

Depending on the accuracy of the results of the algorithm user 114, 116may choose to run the algorithm multiple times to refine and improve thevirtual detector parameters. Reducing both span and step values witheach iteration will yield progressively improved results.

The applied algorithm of FIG. 17 (Step 234 of FIG. 10) not only has theflexibility to lock certain detector parameters in place, but adjust theother parameters to make the model as accurate as possible.

The results may also be adjusted such that if user 114, 116 determinesthat the parameters such as detector height and detector location werenot calculated accurately, the user has the option of specifying theheight, or moving the detector to a more appropriate location. Thealgorithm above is then instructed to not modify the detector height andlocation while iterating. Specifying such known parameters reduces thenumber of unknown variables, detector parameters, and tends to improvethe accuracy of the derived virtual detector parameters.

Once the virtual detector parameters have been determined, it ispossible to project any point in the view of detector 128, 130, meaningany point appearing in image 129, to its respective point on a flatplane and vice versa. Unfortunately, the terrain of site 104 thatdetector 128, 130 may be observing can rarely be simulated by a flatplane. As such, all identified points are assumed to have a z coordinateof zero, and any observed point above a flat plane having a non-zeroz-coordinate would be located incorrectly when mapped to a flat planeimage, such as geospatial map 154. As such, virtual detector parameters,along with known reference points define a virtual 2D detector modelthat is capable of projecting points appearing in image 129 to a flatplane.

Referring now to FIG. 18, for example, detector 128, 130 includes image129 of a portion of site 104. Image 129 displays a hill 292 thatincludes a front face defined by points M-N-L. If an object was detectedat point N in detector image 129, point N projected out onto flat plane290 would not take into account hill 292 that is in the way and wouldcalculate a geospatial location, or set of GCs, incorrectly. Point Nwould be projected onto flat plane 290 incorrectly at point K, when itshould have been projected onto flat plane 290 at point O.

However, by adding points L and M with non-zero z coordinates,interpolation can be used to calculate an intersection of the rayoriginating at detector 128, 130 out to point K with the line segmentM-L, and a more accurate geospatial location of point N can be found atpoint O.

Referring to FIG. 8, three-dimensional reference points viewed bydetector 128, 130, such as point M from the above example, areidentified and stored as indicated at Step 216.

Referring to FIGS. 18 and 19, the details of Step 216 are depicted anddescribed in Steps 294 to 302 as follows.

First, according to Steps 294 and 296, user 114, 115 identifies andassociates more reference points from the detector view, namely pointsfrom image 129, to geospatial map 154 in areas where the elevationvaries. The process is substantially the same as described above withrespect to mapping “two-dimensional” points A-E to geospatial map 154.Points, such as points M and L, are selected at the bottom and top ofelevation changes. Points between M and L will be identified throughinterpolation.

At Step 298, because the position of detector 128, 130 in terms of GCsis known via processes described above, distances from detector 28, 130to each newly entered point may be calculated. Using the virtual 2Ddetector model to convert x,y image pixel coordinates of each point intopan/tilt angles, the straight-line distance from detector to each pointcan be calculated as:Straight-light Distance=Map distance/sin(Tilt)*sin(90 degrees)

3D coordinates are then calculated and added to geospatial model 154using spherical to Cartesian coordinate conversion:x=r sin θ cos φy=r sin θ sin φz=r cos θWhere r is the straight-line distance to each point and theta and phiare the pan/tilt angles of each point.

Though the 3D points calculated are in real-world units, they are in acoordinate space relative to the detector used, and not in terms ofgeospatial coordinates, GCs, used to define a location on geospatial map154. They can not yet be related to other detectors. These points willbe referred to as local points.

When enough local points have been added so that all major elevationchanges have been included and the desired coverage area has beensurrounded, the final step in the process of created a virtual 3Ddetector model 160 is to create a wire frame model 310 of the pointsthat represent the terrain of the viewable area of site 104.

Referring to FIGS. 20-21 a method of the present invention, identifiedpreviously as Step 218 of FIG. 8, uses a combination of well-knownalgorithms to produce a mesh of triangles defining wire frame model 310.Representing the model using triangles provides an easy and lessCPU-intensive method of calculating intersections from detector 128, 130into virtual 3D detector model 160 and linear interpolation of values ofpoints within the triangles.

To create wire frame model 310, first, a Voronoi diagram of the localreference points, for example point N, is created, thereby generatingThiessen polygons for each local reference point. Thiessen polygon 312for point N is depicted, for example, in the Voronoi diagram of FIG. 20.

As will be understood by those skilled in the art, a Thiessen polygonnetwork may be created by employing Delaunay triangulations of the localpoints to generate an intermediate mesh. The Voronoi Diagram is thengenerated by connecting the circumcenters of each triangle around theinput points to create Thiessen polygons.

Next, for each point on the Thiessen polygon, the known technique ofNatural Neighbor Interpolation maybe used to determine z coordinates forthe local points. A final triangulated mesh by connecting the inputpoints with the new Thiessen polygon points into smaller triangles. Suchtechniques are described, for example, in Sibson, R., “A BriefDescription of Natural Neighbor Interpolation,” Chapter 2 inInterpolating Multivariate Data, John Wiley & Sons, New York, 1981, pp.21-36, the contents of which are hereby incorporated by reference.

As such, a three-dimensional wire frame model 310 of the view of eachdetector 128, 130 is created as part of an overall virtual 3D detectormodel 160.

Referring to FIGS. 21-22, a method of mapping, or projecting,three-dimensional local points of detector model 160 to geospatial map154 is depicted and described.

At Step 340, detector focal length is calculated using the horizontalfield of view. In some embodiments, lens distortion may then becorrected at Step 342 by adjusting the screen x,y coordinate inward oroutward from a distortion center point by the distortion magnitude ofdetector 128,130 in detector model 160.

At Step 344, the resulting x,y is then projected into 3D space using thepin-hole camera projection formula. According to Step 346, to adjust forthe detector head's tilt in its detector housing, the x,y,z point isrotated by the detector rotation angle.

Then according to Step 348 to adjust for detector 128,130 not beingmounted level, rotate the x,y,z point by the tilt amount at the tiltdirection angle.

Next, according to Step 350, the local point is projected to adetector-specific flat plane. This is done by first intersecting a lineincluding the x,y,z and detector 128,130 location 0,0,0 with a flatplane positioned at a distance below detector 128,130 location equalingthe detector height, He.

Then, according to Step 352, the location of the local point iscalculated on, or projected to, geospatial map 154. This is done byfirst searching through the triangles in the detector-specific 3D model160 of the viewable area and finding the triangle that intersects with aray that starts at the detector location 0,0,0 and extends out towardthe x,y,z coordinate of the input point. Then, using that intersectionpoint, find its Barycentric coordinates within the intersectiontriangle. Then use the Barycentric coordinates to map the intersectionpoint in detector model 160 to the corresponding triangle on geospatial154 to find the GC.

As multiple detectors 128, 130 are added to site 104, each one goesthrough an identical process building its virtual view of the site intoits model 160. The common x,y,z points that each detector can detect arematched up for all detectors 128, 130 in the model.

By creating virtual detector-specific models 160, identifying locationsof objects 166 in models 160, then projecting these locations on togeospatial map 154, system 100 of the present invention providesfeatures and functions previously unavailable in known systems. As willbe described in further detail below, the unique features include theability to immediately know, and display if desired, the exact distanceof an object 166 being viewed by a detector 128, 130, to the viewingdetector 128, 140. System 100 also provides the ability to pinpoint theexact geospatial location of object 166 relative to site 104 whileobject 166 is in motion and being viewed by a detector 128, 130.

Further, by creating and linking multiple detector models 160 to asingle geospatial map 154, objects 166 may be tracked continuously, orseamlessly. This differs from known systems which do not link multipledetectors to a common geospatial map. Such systems track motion indiscrete zones corresponding to the fields of view of PTZ cameras. Asobjects leave such zones, they are “lost”, then they reappear as newobjects entering the fields of view of another camera set.

Referring again to FIG. 2, once virtual 3D detector models 160 have beencreated and calibrated to geospatial map 154 as depicted by Step 158 andas described above, system 100 of the present invention monitors site104, in accordance with Step 162. If an object 166 in motion is detectedaccording to Step 164, a series of Steps 168 to 170 take place toconfirm or validate motion, locate and track the moving object, anddefine the properties of the object.

Referring to FIGS. 23-36, system 100 of the present invention, insummary, analyzes background and foreground pixels from sequences ofimages 129, then groups blocks of pixels that indicate motion intorectangular regions, motion rectangles based on a rule set. Regions ofinterest are created to keep track of constantly changing motionrectangles. Validated regions of interest on to geospatial map 154, andthe regions may appear to user 114, 115 on terminal 116, 117 as movinggraphical images surrounding an object 166 in motion.

In the context of the present invention, a single digital image 129, orframe, as captured by a detector 128,130, may be defined as a number ofpixels P_(a,b) arranged in a two-dimensional array P of pixels having arows and b columns. Further, and as described below in detail, a singlepixel, P_(a,b,) is associated with a one-dimensional pixel historyarray, HP_(a,b) (see FIG. 24) which includes individual pixel P_(a,b)data for a series of images 129 rendered over a specified time period.

Referring to FIGS. 23-24, to begin the motion detection and validationprocess, and with reference to Step 360, a first, single image 129 froma detector 128, 130 is captured and a pixel array P, having a rows and bcolumns, is formed. Pixel data includes at least red, green, blue (RGB)intensity values.

At Step 362, pixel data for an individual pixel P_(a,b) is sampled attime t_(n), followed by a determination at Step 364 of the RGB intensityvalue for each pixel at time t_(n).

At Step 366, the RGB intensity value for each pixel is converted to asingle 256 bit grayscale intensity value.

At Step 366, the grayscale intensity value is stored into a pixelhistory array HP_(a,b).

According to Step 370, Steps 362 to 368 are repeated for each pixelP_(a,b), until a series of pixel grayscale intensity values I_(a,b)corresponding to a series images 129 to define a pixel array HP_(a,b)for a defined time period and number of samples N.

At Step 372, an initial background intensity is calculated as an averageof the sampled intensity values stored for that particular pixel inpixel history array HP_(a,b). At Step 374, span S, discussed in furtherdetail below, is set equal to a standard deviation σ₁ of the pixelintensity values I calculated for each pixel. Average intensity values Iand span S are stored as metadata in pixels for history array HPa,b atstep 376.

As indicated at Step 378, Steps 362 to 376 are repeated such that apixel history array HPa,b is created for each pixel Pa,b.

FIG. 24, depicts the general structure of pixel history arrays HP_(1,1)to HP_(a,b) for pixels P_(1,1) to P_(a,b). As depicted, each pixelhistory array HP_(a,b) includes N intensity values P_(a,b)I_(a,b) foreach pixel P_(a,b), an average intensity value P_(a,b)I_(avg) for eachpixel P_(a,b), and a span S for each series of N samples of pixelintensities I_(N).

FIG. 25 depicts a series of pixel history arrays for N=10 samples, andin accordance with FIG. 24. In some embodiments, several seconds worthof samples are recorded, thereby capturing data from approximatelytwenty image 129 frames.

The above described series of pixel history arrays HP define an initialbackground B for a particular detector 128, 130.

Similarly, background B may require periodic updating due to otherfactors not associated with motion, including, for example, changinglight conditions, objects added to the viewed image, and so on.

However, because each pixel P_(a,b) may have a different rate ofintensity fluctuation and standard deviation due to image sensor noise,background noise, rippling water, shimmering reflections, etc.,background B must be continually monitored and intensity I_(avg) andspans S updated as needed.

During the first several seconds of flowing images 129, every pixel_(Pa,b) of background B is in “learning mode”. As new images 129 arestored, new pixel intensity values P_(a,b)I_(a,b) are added to the endof each pixel history array HP_(a,b) in background B. Span S for eachpixel P_(a,b) is then calculated based on a multiple of the standarddeviation σ₁ of the pixel history. While in learning mode, span S issaved to background B on each new image 129. Learning mode is done whenit is determined that the pixel history has become “stable” and then thespan is no longer updated for every image.

Because a scene may change, due to lighting, or in the case of a primaryPTZ detector 130, due to the detector moving to a new position, thebackground pixels must be continually updated to accommodate this.Therefore, after establishing an initial Background B in accordance withFIG. 23, in order to update background B, pixel intensity valuesP_(a,b)I_(a,b) are continually updated and stored in pixel historyarrays HP_(a,b) as a detector 128, 130 provides each new image 129.However, pixel intensity averages I and spans S for an individual pixelP_(a,b) are only updated when none of the stored samples P_(a,b)I_(a,b)in pixel history array HP_(a,b) is outside of an allowed intensity rangeR for that given pixel. Range R is defined as average intensityIavg+/−Span S. Any pixel intensity I that falls within range R isconsidered normal and not indicative of motion. Pixel intensities Ioutside of range R are considered spikes, and may indicate the presenceof motion, and therefore should not be incorporated into background B.

Referring to FIGS. 26-27, Steps 380 to 392 describe the process ofupdating background B.

In one embodiment, after defining background B in terms of a series ofpixel history arrays HP_(a,b), which include pixel intensity averagesI_(avg) and spans S, a current, or foreground, image 129 is captured bydetector 128, 130. Then, in order to determine which pixels P_(a,b) havechanged significantly, thereby indicating motion, pixel intensities ofthe current image 129 are stored temporarily as foreground image arrayPF, and compared to stored background B intensity averages I_(avg) andspans S.

Each pixel PF_(a,b) in this new image 129 has a pixel intensityPF_(a,b)I_(a,b) and in one embodiment, is subtracted from the backgroundpixel intensity P_(a,b)I_(a,b). If the magnitude of the differencebetween foreground pixel intensity PF_(a,b)I_(a,b) and background pixelintensity P_(a,b)I_(a,b) is greater than the magnitude of span S, avalue equal to the magnitude of (P_(a,b)I_(a,b)−PF_(a,b)I_(a,b)−S) isstored in a temporary motion image array MI. Otherwise, a zero isstored.

For example, if new foreground pixel intensity PF_(1,1)I_(1,1) is 150,the background intensity P_(1,1)I_(1,1) is 140, and the allowable spanis 20, then the pixel value written to the temporary motion imageMI_(1,1)I_(1,1) is 0, meaning no change at all. If PF_(1,1)I_(1,1) is170 a value of 10 will be stored for MI_(1,1)I_(1,1). Non-zero valuesindicate motion. FIG. 28 depicts an example of a 10×10 motion imagearray that includes multiple of motion pixels.

FIGS. 27 a-d illustrate the updating process as just described.

In some embodiments, using well-known techniques, a Gaussian filter,such as a 5×5 Gaussian filter, may be executed over the motion imagearray MI to create a filtered motion image array, FMI. In otherembodiments, the improvement gains due to filtering may not be requiredfor a particular application.

Additionally, foreground image 129, for example, at time t=0, may alsobe compared to a previous image 129, for example, at t=−1, using themethods described above to compare foreground image 129 to background B.This provides an additional temporary image array showing motion betweenjust the current image 129 and the previous image 129. Such an array isdefined as last motion image LMI. A filtered version of LMI isdesignated FLMI.

Examples of a last motion image array LMI and a filtered last motionimage array LMI are depicted in FIGS. 30 and 31.

Referring again to FIGS. 29 and 31, motion image arrays, either LMIs orFLMIs include differential pixel intensity data as described above. Asdescribed above, pixel cells with non-zero values in motion imagearrays, “motion pixels”, indicate possible motion at a pixel location inan image 129. In some embodiments, a second, simple filter may beemployed to reduce the number of motion pixels by requiring the motionpixel value to be above a pre-defined threshold value, therebydecreasing the probability of false motion.

Adjacent motion pixels may be grouped together, as indicated by shadedborders around the motion pixels of motion arrays EMI and FLMI, asdepicted in FIGS. 29 and 31, respectively. These groups of motion pixelsform motion blobs 400, and in some embodiments may be displayed on ascreen overlaying image 129, and including motion information region 399and motion path 401, for viewing by user 114, 115, as depicted in FIG.32.

While motion blobs are being created, additional information is beingstored about each motion blob 400.

First, Sobel edge detection is used on each motion blob 400 pixel todetermine if the associated foreground pixel is an edge, and if theassociated background pixel is an edge. A motion blob 400 pixel isflagged as a “foreground edge” if the foreground pixel is an edge, andthe associated background pixel is not an edge, or vice versa. The pixelis flagged as a “new foreground edge” if the pixel is a foreground edgenow, but was not a foreground edge in the previous image 129.

A “foreground edge count” is incremented for motion blob 400 if theassociated foreground pixel is an edge. A “new edge count” isincremented for blob 400 if the pixel is flagged as a new foregroundedge. A comparison of the new edge count to the foreground edge countprovides a good indication of how much the actual content of blob 400 ischanging versus uniform changes in pixel intensity, independent of thesize of the object in the camera image. In other words, if a substantialamount of pixels in blob 400 are changing enough to continually createnew edges and possibly cover old edges, it is much more likely there isan object 166 of importance identified in motion blob 400.

Motion blobs 400 may be transformed into one or more types of motionrectangles 402: a base motion rectangle 402 a, an edge motion rectangle402 b, or a difference motion rectangle 402 c. Similar to motion blobs400, motion rectangles 402 indicate areas of motion, and may bedisplayed as a moving graphic overlaying an image 129 for viewing byuser 114, 115.

A base motion rectangle 402 a is stored encompassing the entire blobarea, and is used to describe the position of the blob. FIGS. 33 and 34depict base motion rectangles 402 of motion arrays FMI and FLMI,respectively.

An edge rectangle 402 b is stored only encompassing the foreground edgepixels of its corresponding motion blob 400. Edge motion rectangles 402b are generally preferred over base motion rectangles 402 a to describethe position of a motion blob 400 if there are enough pixels. Thisrectangle typically does not include light shadows that should beignored when calculating the size of a moving object 166.

A difference motion rectangle 402 c is stored only encompassing blobpixels where the current image has changed from the previous image.(determined using the last motion image LMI). Difference motionrectangles 402 c generally may be preferred over edge motion rectangles402 b if there are enough motion pixels available to create asignificant difference motion rectangle 402 c. Difference motionrectangles 402 c are even better for tracking motion and identifyingobjects 166, because they do not include light shadows, and also providean ability to zero in on exactly what is really moving within the entirearea of motion blob 400.

Such methods make it possible for system 100 to discern repetitivebackground motion from true objects in motion, and eliminate falsedetections of motion due to changing background conditions as describedabove.

Even though motion blobs 400 are identified in each new image 129,motion blobs 400 or their motion rectangles 402, can be of any possiblething moving, including left-over image noise, precipitation, subtlelighting changes, etc. Motion blobs 400 may not necessarily represent anentire moving object 166. For example, it may be possible to have oneperson walking, but creating multiple blobs 400 if their shirt happensto be the same color of the background but their head and pantssufficiently contrast with background B.

Therefore, in order to solve this problem, a method of the presentinvention analyzes and correlates motion blobs 400 from one image 129 tothe next, defining regions of interest (ROIs) 404, which may comprisemultiple blobs 400 representing a single object 166 and examines whetherdefined regions of interest (ROIs) 404 are progressive and maintain asteady speed/size.

Referring now to FIGS. 35 a-35 h, series of images 129 at threedifferent points in time are used to depict a method of forming andtracking ROIs 404.

Referring to FIG. 35 a, at time t=0, image 129-0, two objects 166 a and166 b in motion towards tree 405 are detected. Motion blobs 400 a and400 b are defined, as are motion rectangles 402 a-a and 402 a-b.Although any of the three different kinds of motion rectangles 402 a,b,cmay be used, base motion rectangles 402 a are used in this particularexample. Because there initially are no ROIs 404 when motion is firstdetected, new ROIs 404 a and 404 b, equal in size and location to blobs400 a and 400 b, are established.

Referring to FIG. 35 b, at time t=1, image 129-1, objects 166 a and 166b have moved closer to tree 405. New motion rectangles 402 a-a and 402a-b are created to correspond to the new positions of objects 166 a and166 b. ROIs 404 a and 404 b remain available for analysis, and in someembodiments, appear as a screen graphic to user 114, 115.

Referring to FIG. 35 c, each motion rectangle 402 a is compared to eachROI 404 in terms of pixel overlap. The pixel overlap of motion rectangle402 a-a is compared to ROI 404 a to determine that the two have 1,530pixels in common, while 404 a-a has 1,008 pixels in common with 404 b.Similarly, motion rectangle 402 a-b has more pixels in common with ROI404 b.

Referring now to FIG. 35 d, still at time t=1, image 129-1, new ROIs 404a,b are established corresponding to previously identified motionrectangles 402 a,b.

Referring now to FIG. 35 e, time t=2, image 129-2, objects 166 a and 166b have moved, with a portion of object 166 b being obscured by tree 405.In this situation, three motion rectangles 402 a have been created, 402a-a associated due to object 166 a, and 402 a-b 1 and 402 a-b 2 due toobject 166 b. Previously known systems likely would mistake the twoportions of object 166 b, identified by system 100 of the presentinvention as motion rectangles 402 a-b 1 and 402 a-b 2, as twoindependent moving objects 166. Further, a simple pixel overlap test mayalso result in a wrong association of 402 a-a with ROI 404 b. However,the described method of system 100 correctly identifies motionrectangles 402 a-b 1 and 402 a-b 2 as belonging to a single movingobject 166 b, by considering not only pixel overlap, but ROI 404 speedand direction as described below.

Referring to FIG. 35 f, speed and direction of each ROI 404 isconsidered to project ROIs 404 a and 404 b from a time t=1 location toan expected or projected location at t=2, as indicated by the dashedrectangles 404 a-proj and 404 b-proj.

Referring to FIGS. 35 g and 35 h, a pixel overlap is then calculated perthe shaded areas Ap and Bp, using projected ROIs 404, resulting in acorrect identification of 404 a and 404 b at time t=2. This method isthen repeated as necessary to continue tracking objects 166.

This process also ensures the size and position of ROIs 404 will changeand move across the image fluidly.

An ROI 404 is deleted if a frame or image 129 is processed and no blobs400 have been assigned to the ROI. Further, an ROI is initially invalid,and not reported to the geospatial engine.

An ROI 404 becomes pre-validated when all of the following conditionsare true: First, ROI 404 has been assigned blobs 400 for a user-definedamount of consecutive images. Second, the size of ROI 404 has stayedrelatively stable. Stability may be based upon a pre-defined percentageof change allowed for each ROI 404. Increasing and decreasing thepercentage change allowed changes the sensitivity of system 100.

As described above, foreground/background edge characteristics are savedfor all blobs 400. When these blobs 404 are assigned to ROIs 404, thesecharacteristics are also passed on to ROIs 404. An ROI 404 is then givenan object recognition value (ORV) based on a comparison between how manynew foreground edges are in blob 404 versus how many edges alreadyexisted in the area of blob 404 in background B.

Pre-validated ROIs 404 with ORVs that exceed a minimum ORV substantiallyrepresent the spatial area of an object 166 in motion, and are thereforedefined as object regions 406 when projected on to geospatial map 154.Such object regions 406 represent “valid” motion.

Referring to FIGS. 36 and 37, after object 166 is detected, camera imagecoordinates, in terms of pixels, are available for each side of objectregion 406. To determine a size of object region 406, the screencoordinates indicating the location of object region 406, in pixelterms, must first be transformed to three-dimensional detector model 160coordinates. These coordinates combined with virtual detector propertiesof detector model 160 may be used to calculate object region propertiessuch as object region height, width, speed, direction, and so on.

As depicted in FIG. 36, a moving object 166 is defined by its objectregion 406, with defining points P-S, appearing in image 129. Point P isa lower center point of object region 406, Q an upper center point, andR, S are lower corners. Image coordinates for points P-S are known, orcan be easily determined via 3D detector model 160 of detector 128, 130.

To determine the coordinates of object region 406, first project point Pfrom detector image 129 into 3D model 160 using methods described above.Next, find a 3D intersection point of a ray extending out from detector128, 130 at angle α to point Q where the ray meets a distance of D.Subtract resulting z coordinate from detector height H_(c) to get object166 height H_(o).

To get object width W_(o), project points R and S onto detector model160 or geospatial map 154 and calculate the Euclidean distance betweenthem.

As described above, each object region 406 is only projected to detectormodel 160 or geospatial map 154 after it has accumulated a history ofsamples. These samples, in the form of time-stamped data describingobject location coordinates x,y, are used to calculate a real-worldspeed Sp and direction Dr of object 166.

To calculate the angular direction Dr, location data over a specifiedtime period, typically several seconds, is used. Data describing themost current location of the object is excluded. Next, all x,y,coordinates are averaged together to get Xavg and Yavg. Time stamps Tare averaged determine an average time stamp Tavg. Next, calculate anaverage detector angle α′ angle from Xavg,Yavg to most current imagecoordinates to indicate direction.

Speed Sp may be calculated by the following formula:Distance from Xavg,Yavg to current x,y/(Tcurrent−Tavg).

As detector 128, 130 frames or images 129 are processed and new samplesare added to existing object regions 406 on the detector side, thegeospatial metrics of object regions 406 are also calculated and addedto the respective object regions 406.

Such geospatial metrics are not only useful for generally trackingobjects, but may also provide useful, real-time information to user 114,115 who may be interested in filtering or pinpointing objects based onobject metrics. Notably, previously known systems generally do notprovide such real-time metrics as an object is being tracked.

Referring to FIG. 38, after detector-specific object region 406 isprojected to geospatial map 154, system 100 continues tracking objectregion 406. Further, while object 166 via its object region 406 istracked over time, and in the detector view, projected object region 406may change properties while “following” object 166, sometimes disappearand reappearing. This phenomenon appears as periodically changing datawithin detector model 160, and also, in some embodiments, as changinggraphics to user 114, 115 viewing image 129. In one embodiment, user114, 115 may view an image 129 of detector 128, 130 displaying object166 in motion, with a graphic of rectangular object region 406following, or tracking, object 166.

For example, this can happen if object 166 becomes partially obstructed,or especially if fully obstructed, by a tree, car, etc., and reemerges,or if parts of object 166 are same color and intensity as the portion ofbackground B that it passes. In some embodiments, issues of this typeare resolved by the methods described with reference to FIG. 35. Whenobject regions 406 of object 166 are projected to geospatial map 154, itis important to group these multiple paths into one known object usingdetector-specific object grouping so that system 100 knows there is onlyone object 166 to track.

Referring to FIGS. 38 and 39, an embodiment of a process of geospatialdetector-specific object grouping is depicted and described.

Referring specifically to FIG. 38, object region 406A exists at timet-T, where T is a period of time, typically several seconds. Area 408 isdefined by radius R projecting out from object region 406A. Objectregion 406B appears at time t outside of area 408, having traveled alongpath PATH. Object region 406B′, located within area 408, represents anestimated location of 406B had it existed, based upon a projected pathBACKPATH.

Referring also to FIG. 39, the depicted flowchart describes thegeospatial detector-specific object grouping process with reference toFIG. 38.

According to Step 410, site 104 is monitored using detector 128, 130. Ifa trackable object region 406A appears at Step 412, object region 406Ais tracked. If no object region 406 is projected to geospatial 154,monitoring of site 104 by system 100 continues at Step 410. If objectregion 406A continually appears on geospatial map 154 and is thereforetrackable, object region 406A will be tracked at Step 414.

According to Step 416, if object region 406A fails to exist, ordisappears, at Step 418, system 100 checks for a second object region,object region 406B.

If object region 406B exists, it will be tracked according to Step 420.

Next, an actual path PATH of object region 406B is stored and used todetermine an average speed and direction of object region 406B. Theaverage speed and direction of 406B is used to project 406B alongBACKPATH to 406B′, which represents a theoretical location of objectregion 406B at the time that object region 406A ceased to exist, namelytime t-T.

According to Step 424, if object region 406B′ is not within area 408,defined by radius R and the center point of object region 406A at timet-T, the system 100 continues to track object region 406B, assuming thatit is not related to object region 406A. If object region 406B′ iswithin area 408, the width and height of 406A and 406B are compared atStep 426.

According to Step 426, if a height and width of 406A varies from 406Boutside of a certain range, or tolerance, then tracking of object region406B continues at Step 420. However, if the height and width of 406A iswithin a specified range, for example 50%, then object region 406Bbecomes a candidate for actually being object region 406A tracked atdifferent points of time, according to Step 428.

Also according to Step 428, if 406B remains a candidate for 406A for aspecified period of time T, or in other words if object region 406A hasnot been updated for time period T, then according to Step 430, thehistorical data of object regions 406B is merged into object region406A, and object region 406B is destroyed. As such, object regions 406Aand B become one object region 406. In one embodiment, time T is set toapproximately two seconds. If time T is set too low, 406A and 406B maybe incorrectly identified as a single object region 406. Alternatively,if time T is set too high, correlation becomes less likely.

On the other hand, if during time period T, object region 406A“reappears”, then both object region 406B tracking continues accordingto Step 420.

Now that continuous paths of objects are available for each detector128, 130, the next process is to combine each object regions 406 fromeach detector 128, 130 into a single geospatial object group.

Referring now to FIGS. 40 and 41, an embodiment of a geospatialmultiple-detector object grouping method is depicted and described.

Referring specifically to FIG. 40, in this embodiment, three objectregions 406 a,b,c, deriving from three different detectors 128, 130a,b,c, have been mapped to geospatial map 154. The method of the presentinvention determines whether object regions 406 a,b,c are associatedwith a single moving object 166, or multiple moving objects 166 a,b,c,and accordingly forms one or more groups 424 of object regions 406.

Referring also to FIG. 41, according to Step 428, site 104 is monitoredwith multiple detectors 128, 130 a,b,c.

According to Step 429, a first object region 406 is projected togeospatial map 154, and a first object group 424 is created. Firstobject group 424 is substantially the same as first object region 406,until additional object regions 406 are added, or until otherwiseupdated or modified.

According to Step 430, a next object region 406 is mapped to geospatialmap 154.

According to Step 432, any existing groups 424 are checked for recentupdates. According to Step 436, if not updated recently, the projectedlocation is used. Otherwise, according to Step 434, the mapped, ortracked, location is used.

At Step 438, if the distance of a tracked object region 406 is notwithin a distance R of object group 424 for a specified time period T1,object region 406 is not added to group 424. If object region 406 iswithin R of object group 424, then dimensional characteristics of objectregion 406 are considered at Step 440. R and T1 may be determined basedupon desired sensitivity of system 100. In one embodiment, T1 isapproximately equal to 2 seconds.

According to Step 440, if the width and height of object region 406 isnot similar, then object 406 is not added to group 424. In oneembodiment, if both the width and height of object region 406 is notwithin 40% of the height and width of object group 424, then objectregion 406 is similar to object group 424, and is added to object group424 at Step 442.

According to Step 443, if additional object regions 406 need to beconsidered, the steps above are repeated to determine whether eachobject region 406 should be added to group 424.

Furthermore, if an object region 406 matches multiple groups 424 theobject-group pair with the smallest separation distance prevails.

The location, speed, and sizes of the group are determined by a weightedaverage of all its assigned objects. Most recently, updated objects aregiven more weight as opposed to objects that have stopped being updated(i.e. due to occlusion or leaving the view of the detector).

Every group 424 is periodically inspected to ensure that its assignedobject regions from multiple detectors still belong in the same group424. In the case of two people walking down a path and then separatingin different directions, they may be detected as one object region 406in the detector view at first, but then two object regions 406 arecreated when they separate. It is possible that the object group 424 mayincorrectly have the first person from one detector grouped with thesecond person from another detector. This situation is handled by thefollowing:

Gather current locations of each object region 406 contained a group424. If an object region 406 has not been updated recently, itsprojected location is considered instead (based on its last known speedand time since it was updated). If any of the distances between theobject region 406 locations and the group 424 location is larger than Rfor a period of T2 seconds, then the object region is removed from group424. In one embodiment, time period T2 is approximately 3 seconds.Having time period T2 be longer than time period T1 makes it easier foran object region 406 to join an object group 424, as compared to leavinga group 424, potentially increasing the accuracy of system 100.

If an object region 406 is removed from a group 424, it then becomesavailable for grouping again.

Unlike known motion detection systems, by grouping regions of interest406 and projecting these groups 424 onto a common geospatial map 154,system 100 of the present invention can detect and track multipleobjects 166 using a variety of detectors and detection devices, as wellas combine fragments of information, correlate between detectors, andfinally analyze the information at a much higher meaningful level.

In order for system 100 to intelligently decide which groups theavailable primary PTZ detectors 130 should focus on at any moment, theyare prioritized by variety of rules and zones.

Detection zones 460 define portions of geospatial map 154 that need tobe monitored. Zones 460 are typically defined as polygons, but can alsobe poly-lines in the case of detecting when an object passes over a“line in the sand”.

In one embodiment, system 100 assigns a relative priority ranking toeach zone 460. In one embodiment, the priority range has a lower end ofzero, for a least important zone, and an upper end of six, for anextremely important zone. Zone 460 priorities may be used to determine arelative threat of a moving object 166.

In one embodiment, zones 460 also have associated filters 462 thatcontrol what is detected and tracked within its boundaries. Filters mayconsist of minimum and maximum values for object width, height andspeed. Even though all object regions 406 that are projected togeospatial map 154 are grouped according to the methods described above,in one embodiment, only those that pass through at least one filter 462are examined further. Other types of filters may screen for a certainkinds of movement, screen for specific visual characteristics in anobject 166, or in the case of a thermal camera, analyze temperature datato discern unauthorized objects 166.

Referring to FIG. 42, an image 464 of a graphical user interfacedepicting a multi-zone monitored site 104 is depicted. In thisparticular embodiment, site 104 is divided into zones 460 a-460 h. Withthe exception of zones 460 e and 460 h, all zones include filters 462subject to user control via graphical filter buttons 463 and user dialogbox 466.

User 114, 115 may use user dialog box 466 to create customized filters462. Such customized filters 462 may be used to filter, or specificallydetect, certain types of objects 166 or movement. As depicted in FIG.42, user dialog box 466 includes identification entry box 468, objectselection menu 470, and object characteristic entry boxes 472. In theexample of FIG. 42, a filter 462 is set to detect only persons viacriteria such as a minimum height of 1 ft., maximum height of 9 ft.,minimum width of 1 ft., maximum width of 8 ft, and speed up to 25.1 mph.It will be appreciated that any number of filters 462 may be created tofocus system 100 on moving objects 166 bearing any number of detectablefeatures.

Referring to FIGS. 43 and 44, a method of prioritizing detectors 128,130, and validating detected motion is depicted.

Initially, at Step 480, a detection status of system 100 is “false”,meaning that no validated motion is detected. At Step 482, an object 166in motion has been detected in a zone 460, its object region projectedto geospatial map 154, causing the detection status of system 100 tochange to “alert”, according to Step 484. In this embodiment, motion hasbeen detected by a secondary detector 128, rather than a primary PTZdetector 130. Motion detected by a secondary detector 128 must beverified by a primary detector 130 before it is validated. Motiondetected by a primary detector is presumed valid and is processedimmediately according to the methods and systems described above, anddoes not require detection by a second detector 128, 130.

After motion is detected by secondary detector 128, review of availableprimary detectors 130 is undertaken via Steps 488 to 492.

According to Step 486, primary PTZ detector 130 that is the closestprimary detector to the identified GC is located, or determined. If noadditional primary PTZ detectors 130 are available, determination iscomplete per Step 488. If “yes”, the primary detector 130 is checked tosee whether it is active for the identified CC where motion was detectedat Step 490. If detected object 166 in motion may be “seen” by detector130, it may be considered “active.”

If that primary detector 130 is the next closest, and is active in thezone 460 where motion was detected, that primary detector 130 is checkedto see whether the maximum number of validating detectors has beenreached, according to Step 492. An administrator, or user 114, 115 maydetermine the maximum number of primary detectors 130 that may attemptto validate motion detection by a secondary detector 128.

The result of steps 486 to 492 is compilation of a list 500 of availableand active primary detectors 130 that are close to the identified GC.

For example, in the embodiment depicted in FIG. 44, site 104 ismonitored by system 100, which includes three primary PTZ detectors 130a,b,c, and two secondary detectors 128 a, b. Motion has been detected atGC1 and GC2 in zone 460. According to the methods described above,detector 130 b becomes the first primary detector for GC1 primarilybecause it is the closest available detector 130, and becomes the thirdprimary detector for GC2, because it is the third closest to GC1.Similarly, primary PTZ detector 130 c becomes the first primary detectorfor GC1, and the second primary detector for GC 2; primary PTZ detector130 a becomes the second primary detector for GC 1 and the third primarydetector for GC2. In this depiction, all primary PTZ detectors 130 areactive and not requested by other zones

According to Step 496, the GC of detected object 166 in motion is notedand stored, while at Step 488, system 100 creates an available primarydetector list 490. In one embodiment, list 490 would be a list ofavailable primary PTZ detectors 130. In one embodiment, list 490 mayinclude all primary detectors 130, each having a designation open,meaning available, or busy, meaning unavailable.

According to Step 502, all primary detectors 130 on list 498 beginsampling for detection for a minimum period of time. The minimum amountof time may be set by user 114, 115 or an administrator, based ondesired system 100 sensitivity.

If motion is detected at the end of the minimum time period, at Step504, the detection status of system 100 is elevated to “alarm at Step506. Otherwise, if motion is not detected at the end of the minimum timeperiod, the detection status is “false” as indicated at Step 508,causing system 100 to resume normal scheduled activities at Step 510.

According to Step 512, if the detection status is elevated to alarm,primary PTZ detectors 130 continue to stay on target, or track object166.

If, according to Step 514, a primary PTZ detector 130 is not requestedelsewhere, i.e., in another zone 460, that particular primary PTZdetector 130 continues to stay on target in the original zone 460, perStep 502. If that particular primary PTZ detector 130 is requestedelsewhere, a zone priority check is done at Step 516.

According to Step 516, if the new zone 460 is an equal or greaterpriority than the current zone 460, then that particular primary PTZdetector 130 becomes open, per Step 518, meaning it is available for usein other zones, including the new zone 460 as requested. Otherwise, thatparticular primary PTZ detector 130 stays on target per Step 500.

Referring to FIG. 45, the present invention provides systems and methodsfor notifying users 114, 115 of alarms.

In general, system 100 allows users 114, 115 to determine whether toreceive an alarm notice, and how such a notice is to be delivered.

More specifically, according to Step 520, validated motion is detectedand a GC identified. According to Steps 522 and 524, if user 114, 115has elected not to monitor validated motion at the identified GC, thevalidated motion is ignored. If user 114, 115 has elected to monitor theidentified GC, then the notification method is checked via Steps 526 to530.

If user 114, 115 has elected to view validated motion using a cameraviewer, or some sort of system for viewing the images produced bydetectors 128, 130, then an output message 534 is sent via a viewer, oruser 114, 115, interface 536 according to Step 538.

According to Step 528, if user 114, 115 has elected to receive alarmmessages 534 via e-mail, then, according to Step 540, alarm message 534is sent via e-mail server. In some embodiments, images in the form ofescalation sequence imagery 542 may accompany alarm message 534. In oneembodiment, escalation sequence imagery 542 includes sequences of images129 from detectors 128, 130. Further escalation sequence imagery 542 maybe sequenced in a manner that prioritizes images 129 such that the mostimportant images are transmitted first, or otherwise emphasized.

According to Step 544, if user 114, 115 has elected to receive SMSnotifications, alarm message 534 will be sent via SMS service, alongwith images, possibly including escalation sequence imagery 542.

In one embodiment, if user 114, 115 has not elected any methods ofnotification, according to Step 532, alarm messages 534 may be ignored.

In one embodiment, according to Step 546, the success or failure sendingand/or receipt of alarm messages 534 are included in a report.

Various modifications to the invention may be apparent to one of skillin the art upon reading this disclosure. For example, persons ofordinary skill in the relevant art will recognize that the variousfeatures described for the different embodiments of the invention can besuitably combined, un-combined, and re-combined with other features,alone, or in different combinations, within the spirit of the invention.Likewise, the various features described above should all be regarded asexample embodiments, rather than limitations to the scope or spirit ofthe invention. Therefore, the above is not contemplated to limit thescope of the present invention.

For purposes of interpreting the claims for the present invention, it isexpressly intended that the provisions of Section 112, sixth paragraphof 35 U.S.C. are not to be invoked unless the specific terms “means for”or “step for” are recited in a claim.

What is claimed is:
 1. A motion-validating monitoring system fordetecting objects in motion at a site to be monitored, comprising: afirst detector for monitoring a site and transmitting first detectorimage data, the first detector comprising a pan-tilt-zoom (PTZ)detector; and a detector controller adapted to: receive the firstdetector image data from the first detector, and to use the firstdetector image data to create a three-dimensional (3D) terrain model ofthe site, the 3D terrain model defining an x dimension, a y dimension,and a z dimension, the x dimension and y dimension defining a referenceplane of the site, the z dimension defining a distance above or belowthe reference plane of the site, detect an object in motion at the siteby analyzing the first detector image data transmitted from the firstdetector; and track the object in motion within the 3D terrain model,including determining a first location of the object in motion, thefirst location defined by an x, y, and z coordinate of the 3D terrainmodel and including directing the PTZ detector to change a PTZ detectorparameter so as to detect the object in motion at a second location;wherein the creation of the 3D terrain model includes the steps of:identifying a first reference point and a second reference point at thesite to be monitored, the first reference point defined by a first x, y,and z coordinate set, and the second reference point defined by a secondx, y, and z coordinate set, each coordinate corresponding to therespective x, y, and z dimensions of the 3D terrain model; capturing afirst reference image of the site to be monitored using the firstdetector configured at a first known PTZ parameter set that includes apan angle, tilt angle, and zoom level, the first reference imagecomprising a plurality of image pixels, each of the image pixels havingimage pixel data, and one of the plurality of image pixels of the firstreference image being a first reference pixel corresponding to the-firstreference point, such that the first reference pixel also corresponds tothe first x, y, and z coordinate set; capturing a second reference imageof the site to be monitored using the first detector configured at asecond known PTZ parameter set that includes a pan angle, tilt angle,and zoom level, the second reference image comprising a plurality ofimage pixels, each of the image pixels having image pixel data, and oneof the plurality of image pixels of the second reference image being asecond reference pixel corresponding to the second-reference point, suchthat the second reference pixel also corresponds to the second x, y, andz coordinate set; and scaling the 3D terrain model for substantially theentire site, including calculating x, y, and z coordinate sets and PTZparameter sets for image pixels of the first image and the second imagebased on the first and second x, y, and z coordinate sets of the firstand second reference pixels and the first and second PTZ parameter sets,such that image pixels captured by the first detector correspond toactual geospatial points of the site, each of the actual geospatialpoints of the site defined by its respective x, y, and z coordinate set.2. A method of autonomously monitoring a remote site, including:locating a pan-tilt-zoom (PTZ) detector at a detector location at a siteto be monitored; creating a three-dimensional (3D) terrain model of thesite to be monitored, the 3D terrain model defining an x dimension, a ydimension, and a z dimension, the x dimension and y dimension defining areference plane of the site, the z dimension defining a distance aboveor below the reference plane of the site, including: identifying a firstreference point and a second reference point at the site to bemonitored, the first reference point defined by a first x, y, and zcoordinate set, and the second reference point defined by a second x, y,and z coordinate set, each coordinate corresponding to the respective x,y, and z dimensions of the 3D terrain model; capturing a first referenceimage of the site to be monitored using the PTZ detector configured at afirst known PTZ parameter set that includes a pan angle, tilt angle, andzoom level, the first reference image comprising a plurality of imagepixels, each of the image pixels having image pixel data, and one of theplurality of image pixels of the first reference image being a firstreference pixel corresponding to the-first reference point, such thatthe first reference pixel also corresponds to the first x, y, and zcoordinate set; capturing a second reference image of the site to bemonitored using the PTZ detector configured at a second known PTZparameter set that includes a pan angle, tilt angle, and zoom level, thesecond reference image comprising a plurality of image pixels, each ofthe image pixels having image pixel data, and one of the plurality ofimage pixels of the second reference image being a second referencepixel corresponding to the second-reference point, such that the secondreference pixel also corresponds to the second x, y, and z coordinateset; and scaling the 3D terrain model for substantially the entire site,including calculating x, y, and z coordinate sets and PTZ parameter setsfor image pixels of the first image and the second image based on thefirst and second x, y, and z coordinate sets of the first and secondreference pixels and the first and second PTZ parameter sets, such thatimage pixels captured by the PTZ detector correspond to actualgeospatial points of the site, each of the actual geospatial points ofthe site defined by its respective x, y, and z coordinate set; detectingan object in motion at the site by analyzing digital pixel data ofimages of the site transmitted from the PTZ detector; tracking theobject in motion within the 3D terrain model, including determining afirst location of the object in motion, the first location defined by anx, y, and z coordinate of the 3D terrain model and including directingthe PTZ detector to change a PTZ detector parameter so as to detect theobject in motion at a second location.
 3. The system of claim 1, whereinthe PTZ detector captures visual or thermal image data.
 4. The system ofclaim 1, wherein the controller is further adapted to validate theobject in motion at the site by analyzing digital pixel data of imagestransmitted from the first detector to determine if it meets one of acharacteristic of size, speed and uniform motion, relative to thereference plane of the 3D terrain model.
 5. The method of claim 2,wherein the detector is a pan, tilt, zoom (PTZ) camera.
 6. The method ofclaim 5, wherein capturing at least one additional reference image ofthe site to be monitored using the detector includes using the PTZcamera at a zoom setting different from a zoom setting of the firstreference image.
 7. The method of claim 2, wherein tracking the objectin motion within the 3D terrain model includes identifying the locationof the object in motion with respect to the reference plane.
 8. Themethod of claim 2, further comprising validating the object in motion atthe site by analyzing digital pixel data of images transmitted from thedetector to determine if it meets one of a characteristic of size, speedand uniform motion, relative to the reference plane of the 3D terrainmodel.
 9. The system of claim 1, wherein scaling the 3D terrain modelfurther includes calculating x, y, and z coordinate sets and PTZparameter sets for all image pixels of the first image and the secondimage based on the first and second x, y, and z coordinate sets of thefirst and second reference pixels and the first and second PTZ parametersets.
 10. The system of claim 1, wherein the first reference point andthe second reference point share a common x or y coordinate, but do notshare a common z coordinate.